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Introduction 


The field of human factors is as varied and diverse as the human subject itself. But one of its 
most important applications is the facilitation of safety and efficiency in a particular working 
environment through the implementation of paradigms known about the human and their 
working relationship with machines and systems. During the period since World War II (which 
is often viewed as the birth of Human Factors) no area has been the subject of more human 
factors research than aviation. And in no time during that epoch is the influence of human 
factors more important, nor more imperative than it is today. 

As technology driven designs have been finding their way into the national airspace system 
(NAS), there has been growing concern within the aviation industry itself, the Federal Aviation 
Administration (FAA), and the general public for a means by which to certify complex systems 
and the advanced aviation technologies that will be responsible for transporting, directing, and 
maintaining our airborne travel. While it is widely agreed human factors certification is 
desirable, the philosophy that will underlie the approach is debatable. 

There are, in general, two different approaches to certification: 1) the top-down or systems 
approach; and, 2) the bottom-up or monadical approach. The top-down approach is 
characterized by the underlying assumption that certification can be best achieved by looking at 
the system as a whole, understanding its objectives and operating environment, then examining 
the constituent parts. In an aircraft cockpit, this would be accomplished by first examining what 
the aircraft is supposed to do (e.g., fighter, general aviation, passenger), identifying its 
operating environment (IFR, VMC, combat, etc.) and looking at the entire working system 
which includes the hardware, software, liveware and their interactions; then, evaluative 
measures can be applied to the subsystems (e.g., individual instruments, CRT displays, 
controls). 

The bottom-up approach is founded on the philosophy that the whole can be best served by 
first examining it constituent elements. This approach would perform the above certification 
completely antithetically, by looking at the individual parts and certifying good human factors 
applications to those parts under the basic assumption that thewhole is equal to the sum of its 
parts. 
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This paper will attempt to form an argument for the top-down (systems) approach, while 
addressing arguments against it, and pointing out the shortcomings and erroneous assumptions 
inherent within the bottom-up approach. 


Certification 

To develop a cogent argument outlining the advantages of the top-down approach to 
certification, it must first be established what the goals of the certification process are in 
general, and the certification problems that human factors will attempt to overcome. 
Certification, in a generic sense, is the process by which a product is declared appropriate for a 
particular task in that it matches or exceeds a previously defined set of “design to” criteria. In 
being “certified,” it is implicitly understood that the product will safely and effectively perform 
the task for which it was designed. 


Of Aviation Technologies 

Certification of advanced aviation technologies involves, many times, the evaluation of 
products which are technologically new and previously unused or untested. This, in itself, 
poses an interesting problem because with uncharted equipment the standards by which 
previous, like, products had been evaluated now become obsolete and inapplicable. 

Certification of aviation technologies is also unique from some other certification problems 
because of the extensive and unavoidable interplay between many systems, so that the systems 
themselves can be looked upon as subsystems of a larger system. For example, ATC, the fleet 
of aircraft, and maintenance could be seen as systems unto themselves, but on a larger, more 
universal scale, their boundaries are not so narrowly defined; each of the aforementioned 
“systems” are merely players in the entire NAS. Therefore, the certification of each of these 
interrelated entities by themselves falls, trapped, into the quagmire of “fuzzy” certification, 
which is neither desirable nor acceptable. 

The challenge, then, is to overcome these obstacles. The means for doing so appears to be 
the implementation of the systems approach to certification. Because the foundation of this 
theory is built on the premise that the system as a whole is more important than the parts of that 
system, it could be argued that, by its nature, it avoids the aforementioned problems. 


The Systems Approach 

A system can be defined (in a broad sense) as the collaboration of functionally similar objects 
(humans, machines) working towards a common goal within its respective environment. The 
first and most important aspect in designing any system is to clearly define the goals and 
objectives of the system (Christensen, 1987; Meister, 1987). In an automobile, the goal is safe, 
efficient, land travel; in the government, the goal (at least hypothetically) is to serve and protect 
the citizens; and in the airspace system, the goal is to provide safe, expedient, air transport. 


On the Use of the Systems Approach to Certify Advanced Aviation Technologies 


17 


Since the first step in developing a system is the identify the goals, so should the first step in 
certifying that system be to identify the goals, then certify that system based on those goals. 


What Came First: The product or the idea? 

Thomas Edison once said that a new invention consisted of “one percent inspiration and 99 
percent perspiration.’' But, the inspiration, the invention’s objective/goal, comes first. The first 
step in developing any system is to first define what the goals will be; it is impossible, if not 
inconceivable, to begin to build a system without first deciding what it is going to do. The 
Wright Brothers did not just begin to assemble pieces of wood and paper, only to find out, to 
their amazement, that the “thing” they built could fly. 

Just as the goals of the system are a prerequisite for its development, so should they be a 
prerequisite for the system’s certification. The definition of a system’s goals dictates the means 
by which the certification of the subsequent subsystems should be handled. It seems illogical 
and erroneous to attempt certification of products without first considering what the ultimate 
goal of the product is when it is placed in the context of the system. 

Starting with the system’s goal and working down provides a framework within which an 
evaluator can examine the parts as contributing factors towards said goal. This eliminates 
excessive redundancy among components and does not leave room for certain vital components 
to be left out of the system. 


Working Environment 

The systems approach looks at the system and the elements of the system in their working 
environment, and therefore can evaluate the system’s ergonomic layout. By this we are 
referring to the positioning of controls, instruments, displays, etc., within the system and their 
functional relations to each other. This would be similar to a task analysis, where one wants to 
examine the physical relationship of functionally similar, and operationally dependent objects in 
the workplace. This certification is particularly applicable when introducing a new instrument 
or device into a workstation, in which it ends up being placed wherever an opening is available. 

Going from a bottom-up approach, a yoke, for example, could be certified to be poses all of 
the characteristics of a sound human factors yoke, and it is certified on this criterion, but the 
evaluation ends there. The yoke, in all of its glory, is not optimally usable if it is placed behind 
the pilot’s seat by the engineers. It is analogous to writing a book of poetry in Egyptian 
hieroglyphics in the twentieth century; while it may contain brilliant rhyme scheme and flowing 
poetic prose, no one can read it - it is merely wasted paper. Something is only as good as it is 
usable. 

In contrast to the bottom-up approach, a top-down approach would look at a working 
simulation of the aforementioned cockpit, and ergonomic problems like the one mentioned 
above would be recognized. While it is not at all likely that such a huge error would ever come 
about, the point is still valid, and the problem is still real. 

Not only does a system’s ergonomic environment need to be considered in certification, but 
its operational environment needs to be considered as well. By operational environment we are 
referring to lighting, weather, temperature, etc. The minimum light emittance needed for a 
display or instrument is directly related to the environment that it will be use in. Therefore it is 
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necessary to evaluate the instrument while in those conditions. This is not easily accomplished 
through a bottom-up technique, but is easily evaluated within the systems approach. 

It could be argued, by the bottom-up proponents, that the product’s environment could be 
replicated during the evaluation to take into consideration the lighting, for example. But, this 
only really takes care of half of the problem, because another consideration is the light being 
emitted from other displays, the glare, due to the angle of the display in the system, etc. All of 
these environmental factors would theoretically be observed in the systems approach through a 
simulation or mock-up. 


Money, Money, Money 

The major drivers in any system, whether it be in the developmental, evaluative or production 
stages, are cost and cost efficiency. The top-down approach is cost effective in two ways: 1) 
the certification personnel is not required to spend the same amount of time on every product in 
the system because not every part of the system is forced to meet the same criterion of human 
factors engineering; and 2) money is saved in the production of the products because of 
weighted criterion.. 

Toilet Seats and Tool Boxes. The United States Government was under a great deal of scrutiny 
in the mid 1980’ s for purchasing miscellaneous items for its fleet of C-130’s (a military 
transport plane) which appeared to have greatly inflated prices attached to them; some examples 
were $15,000 toilet seats and $5,000 wrenches. The government justified the purchases by 
claiming that the equipment had to be “perfect” in order to be usable and safe in their 
operational environment. While the prices paid for those products were probably justifiable 
(because of the research and development costs for a few production items), it gives insight to 
problems that could be arise out of bottom-up certification: setting outrageously high standards 
for a product before its relative importance in the system and the system’s environment are 
known. 

If every product in the NAS had to be evaluated by the same standards, the prices for the 
products would be exorbitant. No one would argue that the toilet seats on a Boing 757 should 
have to comply with the same human factors standards as the plane’s navigation system, to take 
it to the logical extreme. But, where is the line drawn? How can one judge which products need 
to pass strict human factors and ergonomic tests, without first looking at its role within the 
system. The point is that you cannot. As a result, every product, every display, control and 
widget would be subject to the same meticulous human factors standards. This would result in 
exorbitant prices for the products which would be felt directly by manufacturers and indirectly 
by the paying airline passengers. 

The systems approach would look at the system and the parts that make up that system and 
do something that the bottom-up approach cannot do: decide the relative importance of each 
part, and be able to make a well informed decision as to what standards by which they need to 
be evaluated. So, a rarely used, unimportant product does not have as much time and money 
spent in its certification as a relatively vital, often used product 

In this way, the systems approach would require a reevaluation and alteration of current 
certification standards. Products should to be evaluated on their functional importance: first, 
and human factors standards; second, where depending upon the first cause human factors 
certification may or may not be necessary. For example, if a job required an employee to shovel 
three pounds of coal from the coal pile to the furnace each day, there would be no need to 
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certify that shovel to optimize ergonomic standards. Any shovel from the local hardware store 
would satisfy the requirements sufficiently, and to require anything else would be superfluous, 
and cost inefficient. On the other hand, if the shovel were to be used eight hours a day, five 
days a week then it should be subject to more stringent encompassing human factors standards. 


Workload 

One of the main objectives in any human factors effort is to insure a good fit between the 
humans and the machines they operate. This is done several ways, including personnel 
selection, training, manning, etc.; underlying all of these processes is an evaluation of the 
workload incurred by the human while operating the system, whether it be psychological or 
physiological. 

Workload is a very important aspect of any system design, and it is something that must be 
examined in the certification process. The top-down and the bottom-up approaches address the 
issue from different angles. The bottom-up approach would look at each part of the system and 
measure the workload involved with running that particular part. Then by summing all of the 
measurements, a gauge as to the amount of total workload that would be present in the entire 
system should be had. The problem with this is that, once again, there should be significant 
differences with the parts by themselves and with the human’s management of those parts 
when they are incorporated into the system. This method does not take into account secondary 
tasks. There could be a significantly different amount of workload incurred by the human when 
they are operating the entire system, than the original guesstimation made; and this number 
could be either high or low. Since, neither a high nor low workload is desirable because of 
documented performance deficiencies (Rohmert, 1987), there needs to be a more accurate 
method: the systems approach. 

The systems approach would have the advantage of observing the operator while managing 
the entire system. Subjective and objective tests could be run to determine the amount of 
workload, and appropriate measures could be taken to increase the crew, decrease the crew or 
leave it the same. 


Bottom-Up Approach 


Several of the problems which are inherent within the bottom-up approach have been described 
above. In addition to those problems there are others which not only show this evaluative 
philosophy to be incompatible with certification, but indeed prove it to be undesirable as well. 

Inductive Conclusions 

It has been argued by many well-respected modem philosophers, including David Hume and 
Immanuel Kant, that inductive arguments and assumptions can never be validated. The nature 
of an inductive assumption is that by observing past examples of a particular event, one 
concludes that: in the future, a similar cause will produce the same effect. For example, if we 
drop a penny, at time, t p and it falls towards earth, it will necessarily fall to earth at time, L, . 
According to Hume there are no necessary causal connections, and any attempt to predict the 
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future from the past is a fallacious one that it is built on a circular argument. Kant, not as 
critical, said that there is causality, but we can still never validate a future event based on similar 
past events. 

The process of certification requires us employ inductive logic. We are essentially saying 
that “if X works now at ti, then it will work in the future at t 2 .” Induction is a necessary part of 
certification and cannot be overcome or circumvented. But the number of times that an 
inductive conclusion must be drawn can be minimized. As with all necessary evils, the less the 
better. 

Top-down certification must only use inductive logic once: i.e., in certifying that since the 
system works well during the evaluation, then it will work well in the future (in production 
use). 

Bottom-up certification employs inductive reasoning early as well as later in the certification 
process, thereby making the probability of error more than twice as great. The human factors 
certification personnel must not only certify that part X will work when placed in the system - 
the first case of induction; but, they also must also assume that the system will work when fully 
implemented - the second occurrence of induction. 


Whole * Sum of the Parts 

" ...Because undermining the foundation will cause whatever has been built upon them to fall 
down of its own accord. ” - Ren€ Descartes 

It does not take an advanced degree in engineering physics to deduct that a house that is made 
out of bricks could not be built on a foundation made out of straw, without collapsing under its 
own weight. Basic physics (and common sense) tells us that any physical structure is only as 
sturdy as the foundation upon which it is built. Similarly, in logic, an argument is only valid if 
the premises upon which it is “built” are true. If the foundation is weak or the premises are 
shown to be untrue, the argument crumbles under the weight of invalidity. Bottom-up 
certification is guilty of being built on a straw foundation. 

Bottom-up certification uses as its foundation the premise that the whole is equal to the sum 
of its parts. While this statement may be true with a jigsaw puzzle, it is certainly not true in 
certification, nor any scientific endeavour. As far back as Aristotle - one of the first enquiring 
scientific minds and logicians - it has been recognized that: 

. . . We often fall into error because our conclusion is not in fact primary and commensurate 
universally in the sense which we think we prove it so. We make this mistake when. . . the 
subject (element) which the demonstrator takes as a whole is really only part of a larger 
whole. 

In certification terms, Aristotle would be saying that we often err when certifying a part as a 
entity in and of itself, when it is truly only a part of a larger whole - the system. 

Later in Posterior Analytics, Aristotle’s argument further repudiates the use of the bottom-up 
approach in certification. It says that while a part can be certified by itself the truth of that 
certification is only applicable to the part individually; it would not be true universally, because 
the part is in fact different when it is placed in the system. 

With this weakness exposed the foundation upon which bottom-up certification rests is 
undermined - the theory is invalid. 
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A Bad Product with Good Certification Criteria 

Another, less philosophical, more practical, problem has to do with certification criteria, and 
the role of the certification personnel in the process. It is conceivable for a product to be valid 
by human factors standards without being desirable by them. The three-pointer altimeter 
provides an excellent example (Hopkin, 1994). The altimeter could be certified on the grounds 
that it provides excellent contrast, brightness and font size. It is judged that it could be visible 
from every part of the cockpit, and from a performance standpoint it is accurate to +/- 0.5 feet. 
The problem with this instrument obviously does not within its design, but within the 
instrument itself. 

Incident reports, and experiments analysis of the three-pointer altimeter have shown that is 
responsible for pilot-induced errors a dangerous amount of times. Misreading of 10,000 feet 
are not uncommon. Therefore, perfectly sound human factors instrument, is not a good 
instrument. This once again ties back to the problem of certifying a product, without looking at 
it in its working environment. In the bottom-up approach, this instrument, could be certified; 
not to say that it would not in the systems approach, but it is much less likely. 

A Portrait of the Artist as a Certifier. To illustrate (quite literally) an error that can occur by 
using a bottom-up certification process, I will use an analogy: The Analogy of an Artist as 
Certification Personnel. 

Imagine that you were hired out as a professional art certification consultant. This job 
required that you look at different pieces of art, then certify whether or not they represent what 
they are supposed to (e.g. an eye looks like an eye; a cow looks like a cow). One of your 
clients, a not to bright artist, comes to you and asks you to certify an eye for him which he has 
recently sketched (it looks like Figure 1). The eye looks good - good proportions, proper 
relationship between the pupil, iris, etc. - so you give it your stamp of approval: A good eye. 

Over the course of the next two months the same dimwitted artist shows you another eye, 
then a nose, and then a mouth, all of which look like what they should represent; again the 
obligatory stamp of approval given for each feature. Finally, a couple of weeks later he shows 
you the whole thing, which looks like Figure 2. 



Figure 1 . A good eye 


The picture is poorly drawn, not because any of the parts themselves are poor, nor because 
they features are improperly aligned. The picture is poorly drawn because each feature was 
certified without knowing what the ultimate goal of the painting would be. Each feature, by 
itself is a good drawing (open for debate), and accurately represents the its respective object. 
But, when summed together, the whole is wrong. 
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Figure 2. A bad face. 


Conclusion 


Certification of advanced aviation technologies should not only pose a unique professional 
challenge for the human factors expert as a scientist, but also as a consumer, who wants to 
have the safest air transportation for her and her family. To insure this safety, the best possible 
method of certification should be employed: the systems approach. This is not to say that the 
systems approach is infallible, but it certainly is superior to the bottom up-approach. The 
effectiveness of any certification is only as good as the individual(s) performing it. But, taking 
human error, or misjudgments out of the picture, the systems approach is more sound 
fundamentally, practically and philosophically. 

It is because of its superiority and not it infallibility that the top-down approach is better 
suited to certification. David Hume (1977), philosophical empiricist, argued that human 
judgments and scientific decisions are always made after one entertains two or more opposing 
arguments, examining the possibility of each by weighing the their relative proofs, then 
believing the strongest case. "In all cases we must balance the opposite experiments, where 
they are opposite, and deduct the smaller number from the greater, in order to know the exact 
force of the superior evidence” (p. 74). 

In this situation, the top-down approach provides the strongest case towards its cause; while 
agreeably it brandishes some problems, the positives highly outweigh the negatives. And from 
a Humian approach to decision making should rightly be chosen over its counterpart. 
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